One of the central challenges in computational science and engineering is finding approximate solutions to time-dependent Partial Differential Equations (PDEs). Neural PDE solvers have recently gained traction as they are mesh-free and straightforward to implement. However, backpropagation-based training of neural networks, such as Physics-informed neural networks, often suffers from training difficulties, leading to long training times and limited accuracy. Capturing high-frequency temporal dynamics and solving PDEs over long time spans pose additional challenges. To tackle these issues, we discuss an approach for constructing solutions of PDEs with neural networks trained without using backpropagation. We leverage two core ideas: separating space and time variables and randomly sampling the weights and biases in the hidden layers. Our results demonstrate that our backpropagation-free method surpasses the iterative, gradient-based optimization of physics-informed neural networks in terms of training speed and accuracy, achieving improvements of up to 5 orders of magnitude when applied to challenging PDEs and learning Hamiltonian functions. Our technique may unlock new possibilities for using neural networks as solutions to PDEs, where we specifically address low accuracy and long training times that have historically impeded progress.